Skip to content

feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure#564

Open
leeovery wants to merge 1 commit into
test/prose-cases-start-where-sessions-startfrom
feat/prose-walker-on-sonnet
Open

feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure#564
leeovery wants to merge 1 commit into
test/prose-cases-start-where-sessions-startfrom
feat/prose-walker-on-sonnet

Conversation

@leeovery

@leeovery leeovery commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • Walker moves to Sonnet. Both agents went to Opus back when a walk was judged by the single message an agent returns — a compressed closing summary read as a skipped step, the walker looked unreliable, and the answer was to spend more. With the walk captured turn by turn (feat(prose-tests): judge the walk as told, not the summary returned #561), that reason is gone.
  • Measured, not assumed. A Sonnet walk of implementation-picks-first-task passes every path step with quoted evidence, identical world, no markers. An Opus walk of the same case made the same calls in the same order. Across both cases and both models the code-checked layer came out identical.
  • Sonnet was better on one axis: it declined a system-reminder pushing auto-mode bias against stopping, citing the prose line naming harness auto mode as exactly the thing to ignore. That's the STOP-gate discipline the pipeline depends on.
  • Asserter stays on Opus — the cheaper half (one prompt in, a verdict out) and the half the result rests on.
  • A FAIL is rerun once on Opus by the orchestrator. A failure is where spending more is worth it, and a defect a stronger walker also hits is a defect. Escalation happens in the moment, never by editing the definition, so a routine run is always the model the definition names.
  • Both models are reported on a confirmed or flaky failure, so an escalated rerun is never read as like-for-like.

Test plan

  • Prose suites green (39/39 on corpus + invariants; full prose gate unchanged)
  • The Sonnet verdict this rests on: 5/5 path steps, further claim PASS, world identical: true, all three deterministic checks PASS, zero markers
  • design/prose-tests.md P6 rewritten to match

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564 👈 current
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571

Both agents were put on Opus back when a walk was judged by the single
message an agent returns. A compressed closing summary read as a skipped
step, the walker looked unreliable, and the answer was to spend more.
With the walk now captured turn by turn the reason is gone.

Measured rather than assumed: a Sonnet walk of implementation-picks-
first-task passes every path step with quoted evidence, an identical
world, and no markers. An Opus walk of the same case made the same calls
in the same order. Across both cases and both models the code-checked
layer came out identical. Sonnet also declined a system-reminder pushing
auto-mode bias against stopping, citing the prose line that names harness
auto mode as the thing to ignore — the STOP-gate discipline the pipeline
depends on.

Judging stays on Opus. It is the cheaper half by far, one prompt in and a
verdict out, and it is the half the result rests on.

A FAIL is rerun once on Opus. A failure is where spending more is worth
it, and a defect a stronger walker also hits is a defect. That escalation
happens in the moment, never by editing the definition — so a routine run
is always the model the definition names, and the model of every walk is
recorded and reported either way, which keeps an escalated rerun from
being read as a like-for-like one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant